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ABSTEACT 


Markcv chains with large transition probability matrices 
cccur 1n many applications such as manpower models. Under 
certain conditions the state space of a stationary discrete 
rarameter finite Markov chain may be partitiored into 
subsets, each of which may be treated as a single state ofa 
smaller chain that retains the Markov property. Such a chain 
is said to te “lumpakle" and the resulting lumped chain is a 
special case of more general functions of Markov chains. 

There are several reasons why one might wish to lump. 
First, there may be aralytical benefits, including relative 
Simplicity of the reduced model and development of a new 
model which inherits known or assumed strong properties of 
the criginal model (the Markov property). Second, there may 
re statistical benefits, such as increased robustness of the 
smaller chain as well as improved esStimates of transition 
protakilities. Fania ly the identification of lumps nay 
rrovide new inSignts about the process under investigation. 

However, a problem that arises in connection with frac- 
tical applications cf Markov chain models is to determine 
whether the chain is lumpable. This is especially difficult 
when tne matrix P = {Pi} of tranSition probabilities is 
estimated from transition data. In this case, it 1s desir- 


able to find bounds om A, the Jargest error, > - => eee 


en: 
estimating Fae for (ale ian cage 
This thesis exarines the sensitivity of the lumping 
; : ‘ : : 
conditicns kased on fF, the esStimate of P. In general, it 1s 
found that the classical luamping conditions are extremely 
sensitive to the estimation error which can be expected to 
cccur even with large data sets. Thus, these conditiofrs may 
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I. INTRODUCTION 


Markcev chains with large transition probability matrices 
occur in many applications such as manpower nodels. Under 
certain conditions the state space of a stationary discrete 
parameter finite Markov chain may be partitiored into 
subsets, e€acn of which may be treated as a single state of a 
smaller chain that retains the Markov property. Such a Ghawe 
is said to be “lumpakle" and the resulting lumped chain is a 
special case of more general functions of Markov chains. 

Consider a Markov chain {X:t = 0,1,2,...} with £ ieee 
State space S$ = {l1,Z2,e2-.,n}, Stationary transiticn page 
ability matrix P = (Pid e and a priori distributions 
Winitial states", pe™= (pe, p°0 ieee Let S denote a 
NOnErAVvVial pabticies Of. SS Tito Wie Beit. - say 
S = {1(1),L(2),-.-,L(m)}. If {X,} is lumpable with respege 
to S, denote by (yes) the lumped chain with state space S mame 
transiticn probability matrix BP. 

A well-known characterization [Ref. 2] is that {X,}0as 
lumpable to {X,} if ard only if there exist matrices A and B 
SUCH st rae 


BAPBR = PB (1 Sa 


where B consists of m nonzero orthogonal n-dimensional 
column vectors whose components are zeros or ones, andA is 
E' with rows normalized te probability vectors ({(1.¢é, 
A = (B'*By2B'). The pesitions cf the 1's in each Colunnyoiee 
correstond to states in S$ that together form a lump ir SS. 
It follows that if BAFB = PB is satisfied, then P = APB as 


4S Shown in Chapter 2. 


Many of the mathematical yuantities associated with {X,} 
can ke transformed directly to corresponding quantities for 
(X,}, using the lumping matrix B. In Chapter 2, for example, 
we Show that if an original Markov chain {X,} is lurpable to 
{X,} and ie is ftrrther lumpable to Cane then {X,} is 
directly lumpable to {X,}, and we give the lumping matrix 
BOL (} in terms of the underiying two lumpings. 

There are several reasons why one might wish to iump 
[Ref. 1]. Pie sity, there may be analytical benefits, 
including relative simplicity of the reduced model and 
development of a new model which inherits known or assumed 
strong properties of the original model {the Markcv frep- 
erty). Second, there may be statistical benefits, such as 
increased robustness of the smaller chain as weil as 
improved estimates cf transition probabilities. Peal ly, 
the identification of lumps may provide new insights about 
the process under investigation. 

However, a problem that arises in connection with frac- 
tical apflications of Markov chain models is to determine 
whether the chain is lumpable. For chains with large state 
spaces S, it is practically impossible to use an exhaustive 
search to determine whether lumpability conditions such as 
those given in equaticn (1.1) are met for some matrices B, 
kecause of the large number of ways partitioning S, i.e, the 
large number of candidate B matrices. For example, if S has 
10 elements, there are 115,975 partitions of S. 

Ancther problem is to estimate the matrix p= (Pi) OF 
transition probabilities and to find bounds on A, the 


largest errcr of Ps Pee eddenedts amd) |. We Shall investi- 


' 
gate the sensitivity of the lumping conditions in equation 
Meee LOL Varying A. If {X,} 1s lumpable with lumping 
matrix B, is conditicn (1.1) Satisfied with P replaced by 


the estimate DB? 


This thesis will attempt to examine the sensitivity ot 
the lumping conditions based on reasonable estimation errers 
QZ when P is not knewn and must use estimated by 2a we 
describe these facts about lumpability using eigenvalues and 
eigenvectors, including the theorem mentioned by D.R.Barr 
and M.U.Thomas [Ref. 3}. We do not review elementary 
concerts of Markov chains here; the reader may wish to 
consult [Ref. 2] and [Ref. 4] for review of basic facts and 
specific terminologies such as lumpability, regular Markov 
Chain, etc. 
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II. THEOR 


OF LUMPABILITY 

This chapter will ccver general facts about lumping such 
as, conditions for ijiumping, the number of partiticns 
possitle for any given size of state space S, and theorems 
associated with eigenvector conditions for Markov chain 


fameoabilaty. 


A. CCNDITIONS FOR LUAPING 


Consider as Narkewmenawm [K:t = O071,2,..-} with finite 
Prater space S)=—) {[),2,---,8}, Stationary transiticn prcb- 
ability matrix P= {Pi}, aie TELOCINNGIStriIbutten of 
"initial states", f° = (p®,p%,...,p°). Let S denote a 
nontrivial partition of S into ma<n Wiumps’, that is 
S = {L(1), L(2), ---, L(m)}- If {X,} is lumpable with 
respect to S, denote by {X,} the lumped chain with state 
space § and transition probability matrix P. 

We now show that if the condition (1.1) for lumpability 
with respect to the lumping matrix B, 


BAPB = PR aa} 


mscatesticd, then the lumped transition matrix P is siven 


ry 
P = APB (2.2) 
fe jefoyolree Eis is the sum ) p, , where L(j) is the partition 
k 


barre ; eli) : 
euoSset Ccntaining je S$ ehoel alg LS any element of L(i). By 
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the lumpabibility condition, this value is the same for any 
1 € L(i). But the product PB sums the columns of P in accor- 
dance with the partition subsets indicated by the columns of 
E. Hence, PB is an n xX m matrix with rows repeated in accor- 
dance with the partition sets I1{1), L(2), 2.2.5) (mime 
effect of pre-multipflying by A = (B'B)-*B' is to "average" 
these commcn rows yielding anmxonm matrix P without the 
repeated rows. But such “averages" are just the common Lows 


being averaged. Hence, P= APB is the mx m_ transition 


Mice x of the lumped Craig Wa ea state Space 
{U(1),1(2),---,L(m)}.- 

Example 1. Consider a transition probability  mateiiee 
with 4 states which can be partitioned into 


S = {019 (273), 08) } Se) eee ee 


174 1716 S7%6 72 
P = 0 1712 17 tae sye5 
0 W712 THA 575 

772, V/32, (eee 0 ‘ 


Then 
1 Oa i 0 0 
B= fC es ieee and A = 10  O. 3507550 
0 ee ee @ Cee, 0 ie ae 
Oe. 
Ve know eguation (z.1) ais satisfied with partitioning 


S = fl (1),L(2),L(3)}. Thus, the lumped transition Matra 


0.25 0.25 0.5 
APB = F=1|0 0.167 0.833 
0.875 0.125 0 ? 


Many of the mathematical quantities associated with {X,} 


can ke transformed directly to corresponding quantities for 
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SON ec Werngeacand 5 ct edatwon (2. 1). For example, since 
AB is the m-dimensional identity matrix, it follows that for 


S a positive integer, 


(B) = (APBP = A(PE)A(PB).-.-.A(PB) = AD°B (2.3) 


We now show that (Pf = aApP°B 


—~S 


(P) 


(APB) (APB) (APB) . . . (APB) 
= AF(BAPB) (APB) . . . (APB) 

= AP(PB) (APE) . . . (APB) 

= AP2(BAPB). . . (APB) 

= AP2PB. . . (APB) 


S fpr e 
But AE°B =P , since 
BAPB = PE 
FBAPB = P2B 
BAEBAPB = P2B 


FAP*B = PPR 


Ej) 18) ea JO jc 


so PS is lumpable with the same matrix B and P> = AP°B. 
Bieseiielies im turn that 1f {X,} has steady state distribu- 
Eron {| , then {X,} has steady state distribution = 1B. 


Tneorem 1. The steady state distribution Ty of the 
lumped chain (a is WB where W = TP. 


Proot. TB 


(WP) B 
1B (APB) 
(1 B)P 


Therefore, ly = Ai iee 


ie 


Similarly,the a priori distEtoueeon p® of the initial 
state of the lumped chain corresponding to that Of (themones 
inal chain p°, is given by p? = p°B, since by eguaticns 
(2. 1) Vand 223 )2 


p9p>B = pOp ... PB 


pOP ... PBAPB 
pOP ... PBP 


poOBp*. 


Note that p°P B is the distribution of lumped states cccu- 
pied ty the lumped chain after s transitions. Since this 
equals pOBP® = pops, it follows that po = pOB. 


B. PARTITIONS OF A SET OF SIARES 


The Matrix B consists of m nonzero orthogonal 
n-dimensional column vectors whose components are zeros and 
ones which determine a Specific partition Ox 
SS AMZ yee els Example 1 illustrates this, where the 


state space S = {1,2,3,4} is partitioned into 


S ={{1), (2,3), (4)} = (140; bye eee 


00 
1 QO 
1 0 
1 


cc <_< «as 


0 


Permutations of these cclumns give a matrix which also 


jumps {X,}. In order to see thisjetee B™ be B with cclumns 
permuted in some order. Then BY = BI*, where I is the iden- 
tity matrix with its columns permuted in the same crder. Now 
1f BAFB = PB, then 
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pe pgX = p*(p¥ E*p1B*pB* 
= pre (Bt ar*y1r*st ppr* 
= BI*  (r*y1 (BY By? (1% )-1 1*Bppr™ 
= B(B' BR) BY pBI* 
= BAPBI* 
=e Pee 
= eo 


so it follows that {X,} is also lumpable with respect to the 
matrix B*, 

Now, how many candidate lumping matrices are there? This 
would be the number of partiticns of S. {Ref. 5] gives a 
recursion relation for the number Ay of ways of partitioning 
pest om flpZ gece g N} 2 


N-\ 
N- | 
Ne ( ) aK (N21, A, = 1) (2.4) 
K=O IK 
meOupthic relation we find A,= 1, A,= 2, A,= 5, VS WS, tees 


The sizes of the entries in Table 1 show that it would be 
impossible to use atrial and error approach to finding 
lumping matrices B for lumping a chain with larger state 
Spaces, say with 10 or more elements. Values of A,, for 


larger N are shown in Table 1. 
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mc em er ee cm ee ee ee eee 


TABLE 1 
Partiticns of a Set of N States 


. 
! | 
| N Partit2cns N Partai tienes | 
| =) Se. 20 5 = 1225S x 1013 | 
6 203 a) 8.467490145 x 1023 | 

cd 877 40 1.574505884 x 1035 
8 4140 50 1.857242688 x 1047 | 
| 9 21147 60 9.769393075 x 1059 | 
| 10 115975 70 1.80750039 x 1073 | 
| 
Po EEE EEE _ 


It is of interest to be able to systematically prescribe 
alternative lumpings by generating matrices B for a given 
transiticn matrix P, using some method other than trial and 
error. In the next section, we describe an approach to 
finding #= matrices using the eigenvalues and eigenvectcrs of 
Es 


C. AN EIGENVECTOR CCNDITION FOR MARKOV CHAIN LUAPABILITY 


Many problens in science and mathematics deal with a 
linear operator T : ¥V—-V, and it is of importance to deuew 
mine these scalars for which the equation Tx = Ax has 
nonzerc solutions x. In this section we discuss this 


problem and its relationship with finding matrices B. 


Theorem 2. The value 11s alwavs an eigenvalue fcr any 


Mackov chain transiticn probability matrix. 


Proof. Let ©® be any n x n transition probability Maw 
of {X,}, x be a left eigenvector in R®, and A be the Gemmem 
sponding eigenvalue cf P. Then xP = xy which is equivalent 


to 


x(P -ATI) 0 
For A to be an eigenvalue, 
(225 ).c 


Folution if and only aif 


x of equation Equation (2. 


OF: 


det(P - AI) = 
called the characteristic 
Oe 


matrix 


This is 
A 1 always satisfies eqguaticn 
that the the 

linearly dependent.Ncte that 


columns of 


(oe > Eye f Pp «= +2 
SS ee 26 
P Feee P 
=e) 4) Bae ss : 
p Pp -1] 3 3 e 
| 
Lp Pp e e @ @e +] 
n 
since d, By = 1 for Markov chains, 
J= 
in eguation (2.7) sum to zero, so 
meon2.6) 9 is zero with A= 1. It 


elgenvalue of the Markov chain {X,}. 


properties of 


eigenvectors corresponding to 


(2.5) 


there must be a nonzero solution 


59) will have a nonzero 
(2.6) 

eguation. To show that 
aye, we need only show 
in eguation (2.6) are 
0 ee 
Oo laces. 0 
0 eo 
2 
TO 2 
Sr (227) 

| 
6 De 


it follows that the rews 
the determinant in eyua- 
Bolmisems that A =aMleis an 

We'd next like to see 


the eigenvalue 


A= 1. 

Theorem 3. For any regular Markov chain, components of 
the eigenvector corresponding to A= 1 are proportional to 
the steady state distribution of (X,}- 
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Froof. Let x be a left eigenvector of P, and A be the 


corresponding eigenvalue of P, such that xP = xA, and 
assume 2), xj = 1. For given A = 1, xP = x. The steady state 
AiSheaneios of {X,} is unique [Ref. 4]. Therefore, x aust 
ke the steady state distribution |[[—T since x= 1. 


=| 
The following exarple demonstrates Theorem 3. 


rxamprle 2. Let 


V4 1716. 37 167 

Pp = 0 1712 - 1/12 
0 V7 V28 17 V2 
173 V7 327 Be 0 ° 


The €igenvectors corresponding to the eigenvalues of FF are 


disrlayed as column vectors below: 


Eigenvalues : 1 0) =04 25 moses 
Corser 0 10% > 0.8247 

ELGCCHVEGEORS 4: 0.09209 -0.,7201 =0.375) —07 Gee 
Cs2230 0.7201 -4.,125 =0.7285 
030515 0 =o =O. 509 see 


Note that ae 


( W, e The o Ths a ave ) 
(0.4375, 0.0547, Oeis28. ner se 


0.136 
where Ty, - ee etice 
0.436 + 0.092094 0,22344+ 0,635 


Theorem 4. Eigenvectors corresponding to eigenvalues 
cther than 1 are orthcgonal to e = (1,1,...,1). 


PrOO kh. xe* = x(Pet) = (xP)e' = xAet. Therefcre, xe' 


must kre zero for A # 1. 


we are also interested in finding the relationship 
ketween eigenvalues cf P and those of lumped transition 


BEOrdbl Laity Matar. P, where P = APB as described above. 
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Theorem 5. Suprese {X,} with transition matrix F is 
lumpable to [X,} with transition matrix P. The eigenvalues 


of P are eigenvalues cf P. 


Proof. Let Q(A) = 0 be the (n** degree) characteristic 
equation of P. By the Cayley - Hamilton theorem [Ref. 6], 


EN DEMME Cee) + Ole + "AT = 0 ; 
which together with equation (2.3) implies 


Q,P* + 0. os . + QT 
Q(B) 
0) <e 


AQ(P) B 


Since P satisfies F's characteristic equation and since 
€lgenvalues of Q(P) are of the form QIA), it follows that 
OD ) = 0. Thus all eigenvalues XA of P are also €igenva- 
lues of P. 

We next examine the eigenvectors of P and 2, with the 
aim cf identifying lumpings of {X,} directly in terms of the 
eigenvectors of P. We have seen that po 1s obtained directly 


as pOE; a similar relationship holds with eigenvectcrs cf P. 


Theorem 6. Suppose x is a left eigenvector of P ccrre- 
sponding to eigenvalue A, and suppose {X,} is lumpable to a 
Chain with transition matrix P = APB. Then xB satisfies the 
eguaticn (xE)P = (xB)A. 


Frooti. By equation (2.1), (xB)P = xBAPB = xFE. EUe 
XP = xA, and the result follows. 


We ncete that xB is not necessarily an eigenvector of P 
because it may be zero. In fact, it easily follcws that 
xB = 0 if A is not an eigenvalue of P. But xB may be null 
even if A is an eigenvalue of P, in cases of where A is a 


repeated eigenvalue cf P more times than of ?. 


ye 


[Re£f. 7] pointed out some other useful properties asso- 
ciated with eigenvalues and eigenvectors such as : 1) if the 
Matrix P is symmetric , then eigenvalues are real and eigen- 
vectcrs are different for repeated eigenvalues, 2) if the 
matrix is not symmetric, then the eigenvectors are the same 


for repeated eigenvalues. 


Theorem 7. If {%,} With transition matrix P is lumpakle 


to {X,} with transition matrix P, and ae is lumpable to 
{X,} with transiticn matrix P, then {X,} is direcu 


lumpakle to (X) where (X} is the lumped chain of (xa 


eZ! 


Froot. Let {X,} -)} “Be 


lumpakle tc (ies by matrices B,and B,, where B, and B, are 


be lumpabdle to ONT and { 


lumping matrices in which the dimension nh xX M Of Ble 
greater than that of BL. By equation (2.1), P= A, PB, and 
e = A, PB. Thus, 


aa’ 
F otnen tll 


P =A,PB, = A,(A,PB,)5,= (A,A,) P(E, Be) 


To see that ByB., isa lumping matrix and AsAy)1is "Cie 
required forn, we need to show that (A,A, )-(B,B,) is the 


identity matrix as mentioned in Section A. But 


(AZA,)-(ByB,) = Ay(A;B,)Ba = A;D°E, =0nes, ee 


a a 


Also, note that ByB, is B, lumped by B,, so 2,;B, has columns 
cf the reguired forn. Therefore, {Xj ,} 1s directly lumpable 
to CFG? by the lumping matrix B = B,B, . 


Eon ewes. GonsidersammadnkoOv Chain with 5 States, and 


feansitLen Probability matrix 


OFsee0. 1 Coc eget “O23 
QOnmee0. 3 O. 1 MGe3 0.2 
P = Os Sy 0 ae 0 Ol O23 
Jeu weOez. O.1 9 OR 1 
50.5 CealmeUie2)) Ol ria. 
First, consider S$ = {1,2,3,4%,5} which can be partitioned to 


i (1}, (2-43, (9,93) = {L(1),1L(2),1(3)}. The corresponding 


lumping matrices are 


10 0 
01 0 0 0 
ah = 00 1 and A,= Og S) OSs 
010 0 0 OS) 0 Or Sa; 
Lo, Om 


and the lumped transition probability matrix is 


ts Ore ORS 
ee ney = || 0. 1 ado» 0.3 
0.5 0.2 O23 e 


Secondly, consider S$ with 3 states which can be partitioned 


~~ 


memo { {i533}, {23} = {L' (1),L' (2)}, wath matrices 


B, = 0 1 and A. = O25. 0 oo] 
0 1 0 : 


The corresponding lumsed transition matrix is 
a ns 
BS ee ea ras, ] 
0.4 0.6 gg 
Finally, consider lumping the transition probability matrix 


meeectiy. For partitioning, 
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F = ((1,3,5}4 (2-4}} = (L"(1),1"(2)}, and 


18 enn) 

0 1 OT Ciao 

1 Oj= BeBL= | 0 0 1)/0 1], Aza, = [173 CR See 
Oe O11 Oil 0 O T/27 3072 ’ 
Li Ou Om0 Pw: 


and the directly lumped transition probability matrix is 
> =70.8 0.2 
aa a4 - 

Theorem 7 shows that lumping is "transitive", in the 
follcwing sense. Define two transition matrices P and Q to 
ke equivalent, (P =Q), if Q= 2 Ptomea lumping matrix B 
whose columns are theSe of the identity matrix, in scne 
permuted order. (Thus the chain [X,} and {¥,} differ only in 
the labels associated with their states). Define a relation 
" < " between transition matrices as follows: Q < P 1f and 
only if Q =P for scme lumping matrix 3. Then theorem 7 
shows that 90s P, R<Q » RSP. This relation ~" eee 
reflexive, since ¢ < Q uSing the lumping matrix I (iden- 
tae yee Finally, “<" 1s antisymmetric since Q = Pama 
F<Q-s F =Q. Thus, the set if all transition protability 


Matcices iS partially ordered by the “lumping" partial 


CLASES Sas 
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iry> 


III. BOUNDS CN THE LARGEST ERROR, A, IN 

In this chapter we consider three procedures to find 
bounds on A. First, we use the central limit thecren for 
given i and j. Secondly, we use a binomial approximation on 
the kasis of the first procedure. Finally, we get the 
largest error A, uSing the asymptotic extreme value distri- 
bution. These three approximations are only designed to give 
a rough idea of the relationships between 4 and the number 
M of elements in the state space, the total number of 


cbserved transitions K,,, and the probability &«&. 


Ae APPROACH USING CENTRAL LIMIT THEOREM 


We are interested in the sizes of the errors between the 
estimate P and the unknown P, where P is the transition 
probability matrix cf {X,}- We assume the transition prob- 
ability matrix P is of size M x M. 

Let Rij be the number of observed transitions of {X,} 
from state 1 to state j, and let Ky be the number of 
observed transitions from state i. Similarly kK.) 1s the 
number of observed transitions into state j. 

Lee Py be an unknown transition probability from state i 
to state j and B:, be an estimate of Pi; based on K,, observed 
EbanSitions. Then the usual estimate By of Pj is the ratio 
ona Ry 
Ky 1s fixed, andthe number of transitions from state i to 
state j, Ry , iS Binomial (Rive Py) - Then by the central 
limit theoren, 


IO GOK we i Now, aS a rough approximation, imagine that 
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aA Ky. Pay Cee 
. 2S approximate by NornmalLf p.. —_1 —___== 5 
Ee PP i eee (Ki= 


since 


Kj 
area oe and 


Kak 
YES are) 


< 
ev) 
a 

co 
ae) 

Ss 

ae 

i) 


~ Narl ki] 
~ Csr 


ea 


ue ae Gr re 
~ Kis Pg aoe i3eaR 
Cee 


We want to cind a bound A on the eStimation error 1f,;- EL 
which occurs with prebability at least o~%; that) is; 
largest ZO fom ewhsen 


PLIP- Pyl 2A] 2 o& - 

Now 

P Pi Ms 

PL I Ey- Ppl 2A] = EL ——_---| > ——__—] . (322) 
Ki. Pr; (I~ Pip) Ki-Pej (I~ Pe5) 
Chea YC an 
aa . 

Let i = seksi iia eS , then 
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Z is approximate by standard Normal. Rewrite equation (3.2) 


as 


a 
EL} Z) 2 ————--—-] >a ;3 0 <K&< 1 


Ki. Pos C1— Pi) 


C kar 


Eyguation (3.2) is approxinateiy 


a of 


PrzZ 2 ——=--—-" > 





-~ 


Ki. Pos i> Pi) 
MAP COCK se 


since the Normal distribution is symmetric. Solving for A, 


we nave 


of 
A < w(t - ea) Py tt = Dy) 


y Ki 


‘i guantile of the stan- 





where N1l(1 - ) as the (1 - 





2 
dard Normal distribution. Suppose the steady state frecb- 
aoeiaty |j,.of state i 1s “= based on the equally likely 


case, and suppose the worst case in which 


| 
Best) = 0) Se 
af Pye! ~ B= 


Then an approximate value for A is given by 
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(1 oe 


Na (1 - —=) (Oey 





wid - (0.5) : (3.35 


fp 
“Al 


Equation (3.3) ccncernsethe csnor 1B Py 
and j. We'd now like to find an error bound A’ overall i 


| for fixed 


and j. That is, we wish to find the largest A for which 
ELI By Pi! > f\ for sotle 1 and 7]. 20 


which is roughly the same as 





PL Pi} 2 £\ for some i and J} 2 (3.4) 


a 


We apply the binomial approximation in eguation (3.4), so 
that 


Pl Ey - Fy? Q for some i and jj 


Se PL By - PY < Oo for all awsome 
oo mM 
= 1 — 1 = _—=_ e Bin 
( 3? | ; 


2 


Let I= {1 = i B fer some 0 Ser S 1. Solve for K, whiea 


gives 


=Z- oft -B : (3.6) 


Substitute the value cf ce in equation (3.6) into eguation 


(3.3). Finally we get the approximate bound A for ali i and 
ae 


Ax m( V1 =) (Cay + : (as 


Foguation (3.7) gives an approximate expression for A, using 


binomial approximation. 


Be. APPROACH USING ORTER STATISTICS 


Assume Z,¢ Zy4 «02 « Zm are independent ccntinucus 
random variables, each with density function f, (Zz) and 
distribution F(Z). Now let Zy, 4 Zay ¢ «+2 #& Seqy Aencte 
their ordered values, from smallest 2Z,, to largest Zw) ; 
these are called the order statistics of Zie Lae cee ¢ Lye 


We now consider the frobability law for (Ref. 8j, the 


Jou) 
largest cr maximum value. 


The event {24 5 Z} occurs if and only if the event 


{2,; £ 2, 2, £ Z, --- , 2, S$ zZ} occurs, since if the largest 


Z 1s smaller than z, all M of the random variables must be 
smaller than z, where zis any fixed real number. The 
@estribution function for Z omy is 


F (Ze = Pl 2S 2 
Zia) (M) 


lA 


= P[ Zz, $$ Z, 2, £ Zy wee Z Zo) 


PEG, =) 2) 2i2, = 2) --. P[Z,s 2] 


since Z eZ ,ee+ee2y are assumed independent. But each of Z ,2Z, 


peseeZy has the same distribution F(Z). So 


m 
Eat Z) = ier ce) : 
Zc) Pa 


Zi 


fhe density function sore then is 








d 
i ine ie 
ag i ni 
= —— (eee 
= 4 (PF, (2) Po Bal 
fs d hyiz) 
where £5(2) = ie : 


Consider the lititing distribution fuaction "often 
maximum Zy4) as n tends to infinity. [Ref. 9, 10] show this 


distribu e_oners 


_ a VeigM es, \V 2 og M } 


Liz] F_ (2) a (3 a9 
M-S00 


if Z,, Ze «++ , %Zy is a random sample from standard Nermal 
BOUpWIAG ICI. we want find a bound A on the largest of M2 
errors ketween estimates in ® and the unknown components of 


P. The random variatkles 557 pss are very roughly Normal with 


y 
ri which is derived from eguation 


(See) LCE 2yq f= be Z eee Recall that K,. is the totam 





mean QO and variance 


number of transitions observed. 
Let X = D- . 
random variable X 1s approximately equai to 


where € = 1725... .ue. Then we know the 
| M - 
a ¥ Kes ‘ 


Z has a standard Normal distribution. Let the random vari- 


where 


able Ximay FE egual to max] Py- Pall Then 


aims Pl Xs An) = een P[ maxif.- p.] $ 4 ] 
M-700 mw) M->¢0 yt 


r smallest of Pi - Pj 2-A 
amd 


largest of pis ms Pr; <a 
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Now X,, and X are asymptotically independent, so for large 


M, 


(M?) 


fn Pike =A | 
M20 (m) 


PEUX 2a wma 2A eb Fy, (4) 


2 


[1 - F(a) cr¢ay i 


CH aie (3.9) 


Peem equation {3.9) we derive an expression for A as 


follows. Let A be the largest value for which 


Pl X S07 eS = ae 


(MP) 


This 1s the complementary probability because we wish to 
have FL Ey.- Pyl 2 4 for some i andj] 2 & , as in the 
previous section. The limiting distribution function of the 


maximum X 1s the sameras 


(m4) 


a 


2i¢ ' Kee 2mM~ 
linfF linf F 2A4\— 


en V2 %og2M* ( 24 )-Ee 7 y 2Sog2M" 


it 


C 
Then, approximately, 
— Jrtoaame ( 24 Ee - yalogart 
logii-a&) x -e 3 M 7 


and 


435) 


Log {-leg(1 - & )} #-v/2log2M2 (2 IS -\/2l0g 2M2) . 


Finally, 


Log fo 
~ (0. 1 er (/zlog2M2 - shoe Maks Tz (3. 109 
V2 fog of vee 


Eguation (3.10) is an approximate expression for A kased on 
the asymptotic distrikution of the extreme order statistic. 
We will cecmpare the central limit theorem A's with those 
cbtained with the extreme value distribution, in the next 


secticn. 


C. COMPARISON OF THE THREE EXPRESSIONS 


The three expressions for MA obtained using the central 
limit theorem and order statistics have been developed under 


approximations such as: 1) the steady state distribution 





Of tx} es ~ (egually likely), 2) the variances of 

M 
1B - Fi | have AK. 
all transitions are independent. Information about {X,} is 





as a maximum value (worst case), and 3) 


from the estimate ® because we don't have informaticn about 
the unknewn P. In aview of the above approximations and 
computations, our expressions for A are very rough. dAcwever 
they de provide some insight into the occurrences and ¢1zes 
of estimaticn errors in ?. 

Figure 3.1 contains 3 graphs showing A as a <function c= 
K,, and M for fixed XA = 0.90 kased on the three expressions 
(S25), MSeijerand Weyer. 


The first graph shows error bounds using the central limit 


theorem cn B.- Py for fixed 2 andea,- The second grafh is 
given by the same approach as the first graph, except 
overall estimation errors are considered, for all 1 ana 


a0 
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Figure 3.1 Variation of A for Varying K,, and 4. 


The third graph is Eased on the asymptotic distribution of 


the largest value of IBy- Py 
From Figure 3.1 we see that the largest estimaticn error 


{ over “alli eandese 
depends very much on the number of transition observaticns 


and matrix Size, but not so much on the & value as seen 


£frOn ELgGubews.2 a 


op 


MAXIMUM VALUE OF & 


VARIATION CF A BY CHANG.NG ALPHA 
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Figure 3.2 Variation of A by Changing Alpha («)- 
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Graphs 2 and 3 in Figure 3.1 are very Similar Cvenenomen 
they use different approaches. They give an idea of how 
large jikely values cf A are for given K,. and M, in the 
UWOERStV Case 

If we consider a Markov chain {X,} with M = 20 or 30 
states, and we have observed K = 5000 transitions then, 
ELOUG Aly, it is likely (prob = 07a0) that at least Meme 
element of P is in error by at deast 02 In general, 
expressicns (3.7) and (3.10) may be useful for Markov Gham 
{X 4} with M = 20 or 30 states and large numbers of observed 


transl tprens. 


D. SENSITIVITY OF LUAPING CONDITIONS 


We have developed expressions for 4, using the central 


limit theorem and order statistics. We want to examine the 
sensitivity of the lumping conditions applied to ®, the 
estimate of P. If equation (2.1), which 1S a necessary 


condition for lumping a Markov chain with transition mates 
p, 1s satisfied, then the lumped transition matrix 3 is 
Given, by seq lake LOn (2.2). However, even though P satisfies 
the une inomeconditie ns, it is extremely unlikely that its 
estimate P will also satisfy these conditions, as we shall 
now demonstrate. 

In order to simulate the difference between 9 and 27 
consider a matrix of errors R-A, where R 1S a random matrix 
with dimension the same as P, whose components are 1's, 
-1I's, and O's where the sum of each row iS zero. Now 
consider the lumpability of the simulated estimate vw, which 
1s constructed by taking P plus the random matrix k times 
A, that is, P* = P + R-A. 

Tc show the sensitivity of the lumping conditions, we 
assume the unknown P is lumpable with lumping matrix B, and 
consider the difference (BAP*B - p*p). If equation (2. june 


satisfied Ly P*then all of these components must be zero. 
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Theorem 8. The difference (BAP*B - p*B) is a linear 


function of A. 


Proof. Let R be the random matrix as defined akove and 


let E*® = P + RA. Then (BAP*B - BoB aes given by 


(BA(E+RA)B - (P#tR-A)E} = (BAPE + BAR-A-B - PB - R&-B) 


= (2APB —- PB + (BARB — RB)a& ) 
= (BARB —- RB)-A 
= dey 
Therefore the difference of EAP™B - P*B is linearly depen- 


dent cn A and P* is not lumpakle unless BARB = RB (i.€., R 
is “lumpable"), which 1s not likely to occur. 

Since ® is likely to have elements differing appreciatly 
from the corresponding elements in P (errors of size Qa), it 
can ke seen that tke lumpability conditions will not be 
Satisfied (not even nearly so) by ®, even though {X,} is 
lumpakle. We conclude that attempting to check the Lumpa- 


bility of the estimate ® when P is not known is not useful. 
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IV. SUMMARY AND CONCLUSIONS 

We have given several theorems associated with eigenva- 
lues and eigenvectors for lumpable Markov chains (X,} with 
finite state spaces. We have derived rough, approximate 
mathematical expressicns for the largest error made in esti- 
mating P by ® based cn transition data. 

Both expressions (3.7) and (3.10) are very similar even 
though the estimated MA's for the first expression are 
slightly less than those in the second expression. These 
expressicns show that the largest estimation errors depend 
very Buch on the numkter of transition observations ard on 
the matrix size, but rot so much on the of value. 

Since ® is likely to have elenents differing appreciably 
from the ccrresponding elements in P, it is of ianterestaes 
eXamine whether the eguation BAPB = PB is likely tc be 
nearly satisfied with De 1<C., Wied (BAPB = 2B) be nearly 
zero? ‘This is examined by simulation of "estimates" p* of 
PF, using random perturbations of elements or P of sizes A 
which are likely to cccur as errors in ®. 

This shows that the classical lumping conditions are 
extremely Sensitive to estimation errors which can be 
expected to occur even when a large number of transiticns 
have keen observed. MThus, the classical lumping conditicns 
may be of limited value in many actual applications. 

As further research, it 1s recommended that scme 
constructive approach to finding matrices 3 for lumping a 
dumpakle Markov chain {X,} be developed , perhaps along the 
lines of the theorems mentioned in Chapter 2. It is hored 
that the present study will be useful to those who might 
ctherwise have endeavored to check the classical condition 
for lumpability of a Markov chain {X,} when the transition 


Matrix P has been estimated. 
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